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1.  Introduction 


Teleoperated  systems  development  has  been  increasing  for  U.S.  Army  applications  and  the  Army 
is  continually  attempting  to  field  systems  that  further  remove  Soldiers  from  harm  on  the 
battlefield  (Trouvain,  2006).  For  the  job  of  mine  detection  and  route  clearing,  many  remote 
systems  have  been  designed  and  used,  but  not  many  have  overcome  the  hurdle  of  increased 
Soldier  workload  through  technology.  Chen  et  al.  (2007)  describe  the  various  factors  that  limit 
the  perceptual  and  situational  awareness  of  the  operator  such  as  limited  field  of  view,  limited 
depth  perception,  uncoupled  orientation  and  motion,  control  and  sensor  information  lags,  and 
others.  Muth  et  al.  (2006)  further  describe  the  performance  effects  of  uncoupled  motion  for  both 
command  and  control  vehicles  and  teleoperated  systems. 

Teleoperated  land  systems  for  route  clearing,  mine  clearing,  and  convoy  operations  have  had 
significant  increases  in  attention  by  the  U.S.  Army.  Teleoperation  is  the  remote  control  of  a 
system,  usually  via  radio  link,  that  includes  visual  and  or  other  feedback  to  “immerse”  the 
operator  into  the  remote  environment.  The  task  of  driving  can  induce  high  workload  in  military 
systems  (Wojciechowski,  2004).  Teleoperated  systems,  on  the  other  hand,  are  known  to  have 
significantly  increased  workload  as  compared  to  on-board  or  manned  driving  (Dixon  et  al.,  2003; 
Schipani,  2003;  Scribner  and  Dahn,  2008).  Solutions  to  this  workload  problem  have  been 
sought  in  both  (1)  the  improvement  of  control  and  display  qualities  in  the  operator  station,  and 
(2)  through  the  reduction  of  operator  tasks  through  automation.  However,  the  primary  issue  is 
that  of  loss  of  sensory  information  via  remote  visual,  auditory  and  even  tactile  displays  that 
represent  a  fraction  of  the  information  an  operator  receives  within  the  true  environment.  This 
information-limited  constraint  is  viewed  as  the  primary  source  of  reduced  performance  and 
increased  workload  in  teleoperated  systems.  This  information-limited  environment  is  highly 
analogous  to  an  imposed  “cognitive  tunneling”  effect  described  by  Hancock  and  Warm  (1989) 
and  imposed  by  remote  sensors,  displays,  and  controls. 

This  study  was  designed  in  order  to  assess  the  Soldier  performance  and  interface  quality 
characteristics  of  a  teleoperated  route  reconnaissance  vehicle  system  with  two  teleoperated 
modes  with  respect  to  on-board  driving  as  a  baseline.  Previous  research  has  shown  that 
teleoperation  modes  dramatically  increase  Soldier  workload  and  operational  errors  over  on-board 
driving,  and  a  data-driven  tradeoff  of  removing  Soldiers  from  potentially  harmful  or  lethal 
environments  is  a  strong  consideration  in  the  development  of  these  systems,  which  may  have  a 
high  performance  cost. 

1.1  Teleoperated  Control  Modes 

The  U.S.  Army  Research  Laboratory  (ARL)  has  continually  attempted  to  find  ways  to  improve 
the  Soldier’s  performance  and  survivability  by  leveraging  technology. 
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The  ability  to  control  teleoperated  systems  will  depend  mainly  upon  human  factors  engineering 
(HFE)  interface  design  characteristics,  and  without  suitably  designed  controls  and  displays,  there 
will  be  an  additional  workload  cost  in  such  systems. 

Anecdotal  data  from  a  mine  clearing  system  using  cruise  control  suggests  that  high  workload 
was  attributed  to  the  use  of  cruise  control  in  that  system  (Haas  et  al.,  1997).  Participants  used  a 
joystick  controller  that  may  have  confused  the  operators.  Haas  also  noted  that  HFE  concerns  for 
this  type  of  slow-moving,  sensor-driven  system  would  rely  on  operator  vigilance  to  counteract 
the  high  potential  for  tedium  in  such  a  task. 

1.2  Teleoperated  Systems  Performance  Literature 

Performance  of  teleoperated  systems  is  generally  degraded  as  compared  to  similar  manned 
systems  and  sometimes  half  that  of  on-board  driving  (Scribner  and  Gombash,  1998). 

Specifically,  performance  time  to  complete  a  test  course  for  teleoperated  driving  is  about  twice 
that  of  on-board  driving.  The  number  of  errors  (obstacles  hit)  has  been  shown  to  be  about  double 
that  of  on-board  or  manned  driving  (Scribner  and  Gombash,  1998).  Since  then,  Scribner  and 
Dahn  (2008)  found  that  teleoperator  performance  and  workload  ratings  were  improved  with 
cruise  control,  to  eliminate  the  task  of  continuous  speed  control,  than  without  it.  These  measures 
of  lateral  drift,  or  variance  in  vehicle  position  from  the  lane  center,  may  not  have  been  practically 
significant  and  were  on  the  order  of  a  few  centimeters  in  difference,  albeit  for  very  low  speeds 
only  (Scribner  and  Dahn,  2008). 

The  issue  of  the  source  of  high  workload  under  teleoperated  conditions  is  likely  due  to  a  limited 
sensory  information  condition.  This  limited  sensory  information  condition  has  been  studied  in 
the  past  by  Scribner  and  Gombash  (1998)  and  will  be  discussed  further  in  this  report. 

1.2.1  Motion  Effects  and  Motion  Sickness  in  Telerobotic  Systems 

Motion  sickness  (MS)  is  defined  as  the  physiological  response  of  the  body  when  the  visually 
perceived  movement  and  vestibular  system  sense  of  movement  receive  disparate  cues  of  motion. 
MS  is  often  the  term  used  for  various  types  of  illness  and  has  been  attributed  to  the  body’s  innate 
response  to  neurotoxin  poisoning  (Triesman,  1977).  When  the  vestibular  and  visual  systems  do 
not  have  similar  input,  the  postrema  area  of  the  brain  is  triggered  to  begin  a  vomiting  response. 
This  mismatch  theory  of  system  cues  put  forth  by  Reason  and  Brand  (1975)  can  cause  simulator 
sickness,  created  by  visual  movement  cues  with  a  lack  of  movement  cues,  motion,  or 
seasickness,  caused  by  the  perceived  vestibular  motion  without  visual  input  (Muth  et  al.,  2006; 
Wertheim  et  al.,  1995;  Wertheim,  1998).  MS  in  teleoperated  systems  has  long  been  a  problem 
due  to  both  the  lack  of  visual  fidelity  that  is  associated  with  the  visual  systems  and  the  disparate 
motion  cues  from  either  a  stationary  or  moving  command  platform. 

Tasks  on  moving  platforms  are  difficult,  and  focused  concentration  on  tasks  within  such  systems 
can  elicit  motion  sickness  effects  (Cowings,  et  al.,  1999;  Hill  and  Tauson  2005).  The  Future 
Combat  System  Lead  System  Integrator  performed  a  demonstration  in  which  the  operator 
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teleoperated  robotic  vehicles  from  a  moving  command  vehicle  (Kamsickas,  2003).  The  results 
showed  that  motion  made  all  tasks  more  difficult,  compared  to  an  exercise  in  a  simulated 
environment,  and  some  tasks  (e.g.,  editing  plans  and  maps,  and  target  acquisition)  became  almost 
impossible  to  perform  due  to  the  difficulty  experienced  by  the  operators  in  attempting  to  stabilize 
their  hand  movements.  Operators  also  tended  to  over- steer  their  robotic  vehicles  when  their  own 
vehicle  was  turning  in  disparate  directions  from  the  teleoperated  vehicle  being  controlled. 

According  to  Schipani  (2003),  motion  also  makes  cognitive  tasks  more  challenging.  Schipani 
evaluated  Soldiers’  cognitive  performance  while  in  a  moving  vehicle,  and  found  significant 
accuracy  and  speed  decrements  in  performance.  Degradations  were  found  in  areas  such  as  time¬ 
sharing,  selective  attention,  inductive  reasoning,  memorization,  and  spatial  orientation. 

The  measurement  of  motion  sickness  and  simulator  sickness  can  be  accomplished  with  one  set  of 
measures,  as  the  physiological  outcome  is  the  same  but  can  vary  widely  among  individuals. 
Several  measures  have  been  used  in  recent  literature  concerning  motion  sickness,  but  one  in 
particular  has  been  useful  in  recent  efforts.  For  this  study,  the  MS  inventory  by  Gianaros  (2001) 
was  selected  for  use  because  of  its  ability  to  classify  MS  further  into  specific  areas  of  effect: 
gastrointestinal,  central  nervous  system,  peripheral  nervous  system,  and  sophite -related. 

1.2.2  Workload  in  Telerobotic  Systems 

Performance  and  workload  have  been  assessed  for  many  different  types  of  systems.  It  has  been 
proposed  that  semi-autonomous  modes  of  teleoperated  control  will  yield  the  least  amount  of 
operator  workload.  It  has  also  been  proposed  that  using  cruise  control  will  lower  operator 
workload  as  well.  These  are  all  logical  assumptions  that  remain  to  be  tested. 

Schipani  et  al.  (1998)  found  that  workload  increased  as  mission  distance  increased,  from  500, 
1000,  and  2000  m.  He  also  found  that  workload  increased  as  a  function  of  required  operator 
intervention  in  a  semi-autonomous  system.  The  converse  of  this,  of  course,  is  that  workload 
would  be  lower  for  higher  levels  of  autonomy. 

Glumm  et  al.  (1996)  found  that  when  using  a  computer-aided  teleoperation  (CAT)  method  for 
extending  a  teleoperated  vehicle’s  path,  cognitive  workload  was  increased  in  comparison  to 
direct  teleoperation.  This  may  have  been  due  to  the  distance  between  waypoints,  which  was  set 
at  1  m.  This  rate  of  waypoints  for  the  speed  of  teleoperation  may  have  been  considered  high 
workload.  Speed  averages  were  7.6  and  4.7  km/h  for  normal  and  CAT  modes,  respectively. 

Scribner  and  Dahn  (2008)  found  that  workload  ratings  for  manned  driving,  teleoperation,  and 
teleoperation  with  cruise  control  were  all  significantly  stratified  with  teleoperation  exhibiting  the 
highest  perceived  workload,  ~1.7  times  higher  than  manned  driving,  and  teleoperation  with 
cruise  control  the  next  ranking,  -1.6  times  higher  than  manned  driving  workload.  This  finding  is 
highly  consistent  with  past  research  with  teleoperated  ground  systems. 
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1.2.3  Teleoperation  Performance  Reduction 


No  specific  theory  regarding  the  increased  workload  of  teleoperation  is  available;  however,  there 
are  some  explanatory  concepts  that  have  been  pursued  by  some  researchers.  In  many 
teleoperated  systems,  information  and  control  systems  are  degraded  relative  to  the  operator 
insofar  as  cameras  and  video  links  are  provided  for  visual  and  occasionally  for  auditory 
information.  Control-display  links  for  physical  operation  of  teleoperated  systems  often  have  an 
associated  delay  in  system  response  to  control  actions.  The  first  explanation  of  reduced  sensory 
information  comes  from  older  research  perfonned  by  Scribner  and  Gombash  (1998),  who  found 
that  teleoperation  driving  accuracy  tasks  were  improved  when  a  stereovision  system  was 
compared  to  a  monoscopic  viewing  system.  The  three-dimensional  (3-D)  effect  of  the 
stereovision  system  was  suitable  to  significantly  reduce  road  edge  obstacle  contact  rates  by  up  to 
33%.  A  wide  field  of  view  condition,  using  a  three-camera  array,  was  also  attributed  to  lower 
workload  scores  as  measured  by  the  NASA  Task  Load  Index.  This  evidence  indicates  that 
sensory  infonnation  degradation  in  teleoperation  is  responsible  for  creating  higher  task  workload 
and  stress  conditions  for  comparable  manual  on-board  driving  tasks.  This  is  analogous  to 
driving  in  the  rain  at  night  when  visibility  is  reduced  and  attentional  demand  is  noticeably 
increased  thus  increasing  task  demand  stress  and  workload. 

Some  control  lag  was  evident  in  this  study  but  was  not  controlled  for  in  the  experiment.  This 
data  was  collected  from  a  stationary  teleoperation  platfonn.  It  is  posited  that  a  combinatorial 
effect  of  sensory  degradation,  control  lag,  and  uncoupled  motion  effects  are  the  primary 
contributors  to  increased  task  demand  stress,  increased  task  times,  and  higher  error  rates  in 
teleoperation  environments.  The  typical  reduction  in  perfonnance  and  increase  in  workload  for 
teleoperated  systems  as  compared  to  on-board  or  manual  driving  has  been  known  to  be 
approximately  half  the  perfonnance  (Scribner  and  Gombash,  1998)  and  nearly  double  the 
cognitive  workload  (Scribner  and  Dahn,  2008)  of  on-board  manned  operation,  respectively.  In 
general,  until  systems  are  created  that  can  provide  natural,  high-definition  full  field  of  view, 
immersive  3-D  audio,  near-zero  control  delay,  and  counteract  the  effects  of  uncoupled  motion, 
teleoperated  systems  will  suffer  performance  below  that  of  on-board  driving  in  conventionally 
operated  driving  systems. 


2.  Hypotheses 


We  expected  that  manned  driving,  teleoperation,  and  teleoperation  with  cruise  control  would 
yield  statistically  significant  perfonnance  differences.  Specifically,  we  expected  that 
performance  would  be  significantly  improved  for  manned  operation  mode  compared  to  the  two 
teleoperated  remote  modes.  We  further  expected  that  among  teleoperated  modes,  cruise  control 
would  yield  performance  differences  that  were  significantly  superior  to  teleoperation  mode 
alone.  These  three  modes  are  operationally  defined  in  section  3.3. 
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We  expected  that  subjective  workload  ratings  would  be  substantially  higher  for  both  teleoperated 
modes  as  compared  to  manned  driving  mode.  We  also  expected  that  workload  ratings  for  the 
teleoperation-only  mode  would  be  significantly  higher  as  compared  to  teleoperation  with  cruise 
control  mode. 

We  expected  that  subjective  motion  sickness  ratings  would  be  substantially  higher  for  all  remote 
modes  as  compared  to  manned  driving  mode.  We  also  expected  that  motion  sickness  ratings  for 
teleoperation-only  mode  would  be  significantly  higher  as  compared  to  teleoperation  with  cruise 
control  mode.  We  also  expected  that  motion  sickness  ratings  would  be  significantly  apparent  for 
the  higher  speed  condition. 


3.  Methods 


The  primary  operational  task  in  this  study  was  to  maintain  center  of  lane  position  while  operating 
a  teleoperated  system  at  maintained  speeds  of  8  and  15  mph.  Participants  were  instructed  to 
operate  a  HMMWV  in  manned  or  either  of  two  teleoperated  modes  (operation  modes  =  3). 

These  conditions  were  duplicated  and  identical  under  both  day  and  night  driving  conditions  with 
the  exception  of  different  visual  sensors  that  were  required  for  each  application  and  associated 
level  of  ambient  illumination. 

3.1  Participants 

Participants  were  four  U.S.  Army  Infantry  Soldiers  recruited  from  the  10th  Mountain  Division, 
Ft.  Drum,  NY.  All  participants  met  requirements  for  20/30  visual  acuity.  Ranks  ranged  from 
E-3  to  E-6  (Mode  =  E-4).  Age  ranged  from  22  to  34  years  (M=  26,  SD  =  5.48).  Time  in  service 
ranged  from  1.2  to  16.6  years  (M=  7.1,  SD  =  6.86).  Three  of  the  participants  reported  that  they 
were  low  in  susceptibility  to  motion  sickness  while  one  reported  moderate  susceptibility. 

3.2  Apparatus 

3.2.1  Volunteer  Agreement  Affidavit 

A  Volunteer  Agreement  Affidavit  (VAA)  (appendix  A)  was  given  to  each  test  participant  to 
review  prior  to  participating  in  the  study.  This  fonn  was  used  as  the  single  VAA  for  several 
studies  perfonned  simultaneously  that  were  all  aligned  under  one  research  protocol  number 
ARL-20098-09008,  entitled  “A  Comparison  of  Soldier  Perfonnance  in  a  Moving  Command 
Vehicle  Under  Manned,  Teleoperated  and  Semi-Autonomous  Robotic  Mine  Detector  System 
Control  Modes.”  The  VAA  used  describes  this  study  and  others.  Upon  reading  the  document, 
test  participants  were  able  to  ask  all  questions  concerning  their  participation  in  the  study.  Once 
they  agreed  to  participate,  they  signed  the  document. 
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3.2.2  Demographic  Questionnaire 

A  demographic  questionnaire  (appendix  B)  was  administered  to  collect  age,  gender,  Military 
occupational  specialty  (MOS),  years  in  that  MOS,  and  other  background  information. 

3.2.3  TITMUS  II  Vision  Testing  Device 

Participants  were  screened  for  20/30  both-eye  visual  acuity,  far  distance  using  a  Titmus  II  visual¬ 
testing  device. 

3.2.4  ARL  Robotics  Outdoor  Test  Facility 

The  test  course  is  a  960-m-long  course  that  is  similar  in  nature  to  a  secondary  road.  This  is 
located  at  the  ARL  Outdoor  Robotics  Test  Facility  at  Aberdeen  Proving  Ground  (APG),  MD. 
This  area  has  been  used  in  previous  tests  of  teleoperated  and  autonomous  vehicles  and  operator 
performance.  Vehicle  measurement  plates  were  aligned  and  placed  on  the  center  of  the  road  for 
data  collection  purposes  at  30-m  intervals.  These  plates  also  served  as  visual  identifiers  of  the 
road  center  for  vehicle  alignment.  This  track  was  driven  equally  in  left-handed  laps  for  a  total  of 
four  laps  (3840  m)  or  2.4  mi. 

3.2.5  HMMWV  Teleoperated  Vehicle 

An  armored  HMMWV  was  equipped  to  be  driven  in  manned  on-board  mode  in  normal  driver’s 
seat  with  standard  driving  controls.  This  same  HMMWV  was  operated  remotely  from  the  rear 
right  passenger  seat  of  a  MRAP  vehicle  under  two  different  teleoperated  modes.  The 
teleoperation  operator  position,  controls,  and  displays  within  the  MRAP  vehicle  are  shown  in 
figure  1.  The  HMMWV  to  be  controlled  was  outfitted  with  visual  camera  sensors  for  use  while 
in  teleoperation  mode.  The  sensors  used  were  varied  for  both  day  and  night  conditions.  The  day 
conditions  used  normal  color  day  CCD  cameras  with  a  55°  field  of  view.  The  night  cameras 
were  low-light  monochrome  CCD  cameras  with  a  55°  field  of  view.  These  were  switched  as 
needed  for  day  and  night  operation. 
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Figure  1.  Soldier  teleoperating  HMMWV  from  MRAP  vehicle. 

3.2.6  Teleope rated  Control  Station 

The  Rabbit  Control  Station  was  placed  in  the  rear  of  an  MRAP  vehicle  for  operations  on  the 
move.  The  video  set-up  for  this  study  was  to  use  high  center  camera  view  as  the  analog  display 
(left)  and  a  driver-referenced  center  camera  as  the  redundant  digital  display  (right).  The  right- 
side  display  included  vehicle  status  information  such  as  speed  and  drive  gear  currently  engaged. 
In  all  experimental  trials  except  for  manned  driving,  the  HMMWV  was  followed  by  the  MRAP 
Command  Vehicle.  The  control  station  had  two  LCDs  for  visual  displays  (figure  1). 

The  vehicle  controls  consisted  of  a  steering  wheel  with  dead-man  switches  on  the  back  side  of 
the  wheel  in  case  steering  control  was  lost.  Throttle  and  brake  hand-levers  were  mounted  on  the 
right  and  left  upper  sides  of  the  steering  control,  respectively. 

3.2.7  Workload  Rating  Scale 

The  NASA-TLX  has  been  validated  with  mathematical  processing  tasks  of  various  levels  for 
workload  assessment.  A  modified  version  of  the  NASA  TLX  (appendix  C)  (Hart  and  Staveland, 
1988)  was  used  to  quantify  participant  workload  ratings  under  various  conditions.  The  three 
subscales  for  physical  demand,  mental  demand,  and  temporal  demand  were  used.  A  pair-wise 
comparison  was  used  to  develop  individual  weightings  for  each  workload  aspect. 
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3.2.8  Motion  Sickness  Assessment  Questionnaire. 

The  Motion  Sickness  Assessment  Questionnaire  (MSAQ)  (appendix  D)  (Gianaros  et  al.,  2001) 
was  used  to  quantify  participant  motion  sickness  ratings  under  the  various  conditions. 

3.2.9  Control  and  Display  Ratings  Questionnaires. 

A  Modified  Cooper-Harper  for  assessing  Unmanned  Vehicle  Displays  (MCH-UVD)  assessment 
scale  (Cummings  et  al.,  2006)  was  used  to  assess  the  quality  of  controls  (appendix  E)  and 
displays  (appendix  F)  for  each  condition. 

3.3  Teleope rated  System  Control  Modes 

The  Rabbit  2.0B  System  employed  a  remotely  controlled  HMMWV  commanded  from  an  RG-31 
command  vehicle  using  an  operator  control  unit  mounted  in  the  right  rear  of  the  MRAP.  The 
Rabbit  2. OB  system  had  various  camera  configurations  available  including  a  center-of- vehicle 
mounted  analog  camera  (on  top  of  the  cab),  forward-looking  windshield-referenced  digital 
camera,  and  a  rear-facing  camera  for  reverse  gear.  The  system  was  capable  of  being  operated  in 
one  of  three  modes:  manned,  teleoperated,  and  teleoperated  with  cruise  control.  These  modes 
are  described  as  follows: 

•  Manned  Driving  -  consisted  of  driving  the  HMMWV  control  station  vehicle  around  the 
course  in  the  same  manner  as  the  remote  modes. 

•  Teleoperation  -  the  remote  operation  of  a  vehicle  using  a  video  camera  and  primary  video 
display  in  conjunction  with  a  steering  wheel  and  hand  controls  for  acceleration  and 
braking.  There  is  currently  no  audio  feedback  system. 

•  Teleoperation  with  cruise  control  -  the  previously  described  mode  with  the  addition  of 
continuous  speed  control  via  cruise  control,  essentially  locking  the  desired  speed  and  easily 
disengaged  with  a  cancellation  of  the  cruise  or  with  a  brake  input. 

3.4  Teleoperated  Vehicle  Operation  Scenario 

The  participants  reported  to  APG  and  were  provided  an  overview  of  the  study,  at  which  time 
initial  questions  could  be  asked  about  the  purpose  of  the  study  and  what  was  expected. 
Participants  were  asked  to  read  and  sign  the  informed  consent  form  if  they  agreed  to  participate. 

The  demographics  questionnaire  and  eye  exam  were  then  administered.  Participants  were  then 
familiarized  with  the  purpose  and  method  of  response  for  the  workload,  motion  sickness,  and 
control/display  rating  questionnaires.  Prior  to  any  training,  participants  received  a  safety  briefing 
on  the  operation  of  teleoperated  vehicles  and  ranges.  Participants  were  informed  that  they  could 
withdraw  from  the  study  at  any  time  for  any  reason,  especially  if  they  felt  that  they  had  become 
motion  sick. 
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Following  this  training,  all  four  experimental  conditions  were  presented  to  the  participants. 

There  was  an  optional  rest  period  between  trials,  yet  none  of  the  participants  chose  to  rest  in 
between  trials.  Workload,  motion  sickness,  and  control/display  questionnaires  were  all 
administered  after  each  four-lap  experimental  trial. 

3.5  Design  and  Analysis 

The  design  of  this  experiment  was  a  3  x  2  repeated  measures  design.  Separate  3x2  factorial 
repeated  measures  ANOVAs  were  used  for  both  day  and  night  conditions  data.  The  treatment 
variables  had  three  levels  of  control  mode  and  two  levels  of  speed.  Tukey  least  significant 
difference  (LSD)  was  used  for  pair-wise  post-hoc  comparisons. 

3.5.1  Independent  Variables 

The  first  variable  manipulated  in  this  study  was  the  control  mode  of  vehicle  operation  for  the 
Rabbit  2. OB  system,  which  at  the  time  of  test  included  a  HMMWV  as  the  teleoperated  vehicle. 
Levels  of  control  mode  were  manned  driving  (for  baseline  comparison),  basic  teleoperation,  and 
teleoperation  with  cruise  control.  The  second  variable  manipulated  in  this  study  was  speed  of  the 
vehicle,  which  was  either  8  or  15  mph.  The  time  of  day  was  also  manipulated  and  was  either 
mid-day  from  11  a.m.  to  3  p.m.  or  night  from  9:30  p.m.  to  1  a.m.;  however,  this  last  variable  was 
considered  to  be  separate  due  to  confounding  visual  sensors  and  ambient  illumination,  and  two 
separate  analyses  were  performed  based  upon  the  differences  in  visual  sensors  employed  for 
teleoperated  viewing. 

3.5.2  Dependent  Variables 

Dependent  variables  included  vehicle  performance  data  (lane-center  deviation  and  road-edge 
crossings),  workload,  motion  sickness,  and  control-display  ratings.  Lane-center  deviation,  or 
lateral  drift,  was  measured  on  a  scale  of  inches,  and  the  numbers  of  lane  edge  misses  were 
counted  for  totals.  Lateral  drift  is  a  measure  of  deviation  from  the  center  of  the  lane  as 
determined  by  the  position  of  the  vehicle  passing  over  lane  center  ground  marking  plates,  which 
were  read  by  laser  sensors  as  they  passed  over  each  plate.  Lateral  drift  was  collected  at  each 
ground  marker  plate  every  30  m  and  was  recorded  for  future  analysis.  Road  edge  crossing  errors 
were  recorded  when  the  vehicle  did  not  register  a  ground  marker  sensor  plate  pass,  indicating 
that  the  vehicle  would  have  traversed  to  the  road  edges  or  further  to  either  the  left  or  right  side. 

Subjective  ratings  of  workload  and  motion  sickness  were  also  assessed  using  the  NASA-TLX 
and  Motion  Sickness  Assessment  Questionnaire,  respectively.  Additionally,  all  primary  controls 
(throttle,  brake,  steering  control)  and  displays  (primary  visual  display  and  speedometer)  were 
assessed  with  a  MCH-UVD  rating  scale  inventory  to  present  means  and  standard  deviations  of 
these  elements. 
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4.  Procedure  and  Methodology 


The  participants  reported  ARL  Robotics  test  facility  at  APG  to  begin  study  participation.  As  part 
of  the  pre-test  procedure,  participants  were  given  a  volunteer  agreement  affidavit  describing  the 
study  and  possible  risks,  during  which  time  all  questions  pertaining  to  the  study  were  addressed. 
Prior  to  experimentation,  participants  were  tested  for  visual  acuity  using  a  Titmus  II  vision¬ 
testing  device.  Demographic  data  was  also  collected  at  this  time.  All  participants  were 
familiarized  with  the  workload,  motion  sickness  (MSAQ),  and  control/display  rating 
questionnaires  (MCH-UVD)  immediately  following  visual  screening  and  demographic  data 
collection.  This  was  followed  by  several  days  of  participant  training  on  the  teleoperated  system 
in  both  control  modes  under  day  and  night  conditions. 

Participants  received  3  days  of  system  training  from  system  vendors  with  a  total  of  12-h 
operation  time  teleoperating  the  system  from  qualified  system  trainers.  This  training  occurred 
immediately  prior  to  experimental  data  collection  which  followed  training. 

4,1  Participant  Scenario 

System  familiarization  was  given  to  each  participant  prior  to  the  study  to  ensure  that  proper  and 
safe  operation  of  the  system  would  be  performed.  Participants  were  assigned  to  their  subject 
identification  numbers  and  subsequent  condition  orders  prior  to  the  experimental  data  collection 
day.  The  order  of  presentation  conditions  is  presented  in  tables  1  and  2. 

Table  1 .  Day  -  trial  presentation  order. 


Trial 

1 

2 

3 

4 

5 

6 

TP1 

M8 

M15 

T8 

T15 

TC8 

TC15 

TP2 

M15 

T15 

TC15 

T5 

M8 

TC8 

TP3 

T8 

TC15 

TC8 

M8 

M15 

T15 

TP4 

T15 

TC8 

M8 

M15 

TC15 

T8 

Note:  M  =  Manned,  T  =  Teleoperated,  TC  =  Teleoperated  Cruise  Control,  8  =  8  mph,  and 


15  =  15  mph. 

Table  2.  Night  -  trial  presentation  order. 


Trial 

1 

2 

3 

4 

5 

6 

TP1 

TC8 

M8 

T15 

TC15 

T8 

M15 

TP2 

TC15 

TC8 

T15 

T8 

M15 

M8 

TP3 

M8 

M15 

T8 

T15 

TC8 

TC15 

TP4 

M15 

T15 

TC15 

T5 

M8 

TC8 

Note:  M  =  Manned,  T  =  Teleoperated,  TC  =  Teleoperated  Cruise  Control,  8  =  8  mph,  and 


15  =  15  mph. 
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At  the  experimental  test  site,  participants  were  given  additional  training  and  familiarization  with 
the  remote  control  modes  by  operating  up  to  two  laps  around  the  test  course.  For  each  test 
condition,  the  participants  drove  for  four  laps  as  accurately  as  they  could  while  attempting  to 
maintain  the  designated  speed  using  their  visual  display  to  track  the  vehicle  with  the  road  center. 
All  steering,  braking,  and  throttle  controls  were  performed  with  the  hands  using  the  steering 
wheel  or  controls  located  on  the  steering  wheel.  After  four  laps,  the  participants  were  asked  to 
rate  their  workload  by  assessing  several  workload  subscales,  their  motion  sickness  with  the 
MSAQ,  and  their  quality  ratings  of  the  controls  and  displays  with  the  Modified  Cooper-Harper 
Scale.  Test  participants  were  then  fully  debriefed  and  given  a  point  of  contact  for  follow-up  on 
individual  performance  or  results  of  the  study.  All  participants  in  this  study  were  exposed  to  all 
control  modes.  Following  the  study,  test  participants’  motion  sickness  was  assessed  to  determine 
if  they  should  remain  at  the  facility  due  to  these  effects.  None  of  the  four  test  participants 
exhibited  even  mild  effects  of  motion  sickness  following  the  study. 


5.  Results 


Two  separate  3x2  factorial  repeated  measures  ANOVAs  were  used  to  examine  the  main  effects 
of  control  mode  and  speed,  and  the  possible  interaction  effects  among  mode  and  speed  for  all 
dependent  measures  (alpha  =  0.05  significance  level),  including  lateral  drift  from  the  center  of 
the  lane  (in),  road  edge  crossing  errors,  workload  ratings,  and  motion  sickness.  These  two 
separate  analyses  were  performed  for  separate  day  and  night  conditions.  Tukey’s  LSD  was  used 
as  a  post-hoc  analysis  to  perform  pair-wise  comparisons  between  treatment  cells.  Day  and  night 
conditions  were  analyzed  separately  due  to  the  change  in  visual  sensors  for  day  and  night 
operation. 

5.1  Day-Time  Analyses 

The  results  for  lateral  drift  data  yielded  significant  results  for  the  effect  of  speed,  F(\  ,3)  =  16.38, 
p  =  0.027,  and  r\2  =  0.845.  The  effect  size  was  very  large  accounting  for  84.5%  of  the  variance. 
The  lateral  drift  means  for  each  condition  with  associated  standard  error  bars  are  presented  in 
figure  2.  The  main  effect  for  mode  and  the  interaction  were  both  non-significant.  The  results  for 
road  edge  crossing  error  data  yielded  significant  results  for  the  effect  of  speed,  F(  1 ,3)  =  13.34, 
p  =  0.035,  and  rj2  =  0.816.  The  effect  size  was  very  large  accounting  for  81.6%  of  the  variance. 
The  road-edge  crossing  error  means  for  each  condition  with  associated  standard  error  bars  are 
presented  in  figure  3.  The  main  effect  for  mode  and  the  interaction  were  both  non-significant. 
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Mean  Lane  Position  Deviation  (in.)  by 
Speed  (mph) 

12.00  -| 

10.00  - 
8.00  - 
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2.00 
0.00 
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Figure  2.  Mean  lane-position  deviation  (in)  by  speed  (mph)  -  day  operation. 
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Mean  Lane  Road  Edge  Crossing  Errors 
by  Speed  (mph) 
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Figure  3.  Mean  lane  road-edge  crossing  errors  by  speed  (mph)  -  day  operation. 
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Results  for  workload  ratings  demonstrated  a  nearly  significant  effect  for  mode,  F( 2,6)  =  4.32,  p  = 
0.069,  and  rj2  =  0.590.  The  effect  size  was  moderate,  accounting  for  59.0%  of  the  variance. 
Though  non- significant  at  the  p  =  0.05  alpha  level,  the  explained  variance  is  considerable  and  the 
workload  means  for  each  condition  with  associated  standard  error  bars  are  presented  in  figure  4. 
The  main  effect  for  mode  and  the  interaction  were  both  non-significant.  The  Tukey’s  LSD  post- 
hoc  test  revealed  that  the  differences  were  due  to  cell  differences  between  the  teleoperation-only 
and  teleoperation-cruise  control  modes. 

The  results  for  motion  sickness  ratings  revealed  non- significant  results  for  the  effect  of  speed, 
F(l,3)  =  6.99,  p  =  0.077,  and  r]2  =  0.700.  The  effect  size  was  very  large,  accounting  for  70.0% 
of  the  variance.  Though  non- significant  at  the  p  =  0.05  alpha  level,  the  explained  variance  is 
considerable  with  reasonably  high  power.  The  main  effect  for  mode  and  the  interaction  effect 
were  also  both  non-significant. 


40.00 

35.00 

30.00 

25.00 

20.00 

15.00 

10.00 

5.00 

0.00 


Workload  Ratings  by  Control  Mode 


On-Board  Teleoperated  Teleoperated-Cruise 


Figure  4.  Overall  workload  ratings  by  control  mode  -  day  operation. 


These  results  are  astounding  considering  the  extremely  small  sample  size  (n  =  4)  and  the 
moderate  and  large  effect  sizes  revealed.  The  results  reflected  an  increased  lateral  drift  away 
from  the  road  center  under  the  higher  speed  of  15  mph  as  compared  to  8  mph,  as  collapsed 
across  all  conditions.  A  summary  of  all  ANOVA  data  is  presented  in  table  3. 
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Table  3.  Daytime  ANOVA  table  of  dependant  measures. 


Lateral  Drift  (in.) 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

14.59 

2 

7.30 

0.523 

0.618 

0.148 

0.10 

Speed 

52.42 

1 

52.42 

16.38 

0.027 

0.845 

0.76 

Mode  x  speed 

7.34 

2 

3.67 

1.02 

0.42 

0.253 

0.16 

Road  Edge  Error 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

2000.25 

2 

1000.12 

0.84 

0.478 

0.208 

0.14 

Speed 

6970.04 

1 

6970.04 

13.34 

0.035 

0.816 

0.69 

Mode  x  speed 

196.08 

2 

98.04 

0.40 

0.69 

0.117 

0.09 

Workload 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

950.91 

2 

475.45 

4.32 

0.069 

0.590 

0.44 

Speed 

4.16 

1 

4.16 

0.86 

0.42 

0.222 

0.10 

Mode  x  speed 

11.89 

2 

5.92 

0.39 

0.69 

0.115 

0.09 

Motion  Sickness 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

18.25 

2 

9.12 

0.92 

0.45 

0.234 

0.14 

Speed 

1.04 

1 

1.04 

6.99 

0.077 

0.700 

0.70 

Mode  x  speed 

20.58 

2 

10.29 

2.01 

0.22 

0.401 

0.27 

5.2  Night-Time  Analyses 

The  results  for  lateral  drift  data  yielded  significant  results  for  the  main  effects  of  speed, 

F(l,3)  =  47.87,  p  =  0.006,  and  rj2  =  0.941.  The  effect  size  was  very  large  accounting  for  94.1% 
of  the  variance.  The  lateral  drift  means  for  each  condition  with  associated  standard  error  bars  are 
presented  in  figure  5.  The  main  effect  for  mode  and  the  interaction  were  both  non-significant. 
The  results  for  road  edge  crossing  error  data  yielded  significant  results  for  the  main  effects  of 
speed,  F(l,3)  =  85.47,  p  =  0.003,  and  rj2  =  0.966,  and  mode,  F  (2,  6)  =  8.67  and p  =  017.  The 
effect  size  for  this  result  was  r|p  =  0.743,  with  the  mode  treatment  accounting  for  74.3%  of  the 
variance,  a  very  large  treatment  effect.  The  road  edge  crossing  error  means  for  each  condition 
with  associated  standard  error  bars  for  the  effects  of  speed  and  mode  are  presented  in  figures  6 
and  7,  respectively.  The  interaction  was  non-significant.  The  post-hoc  analysis  for  road-edge 
crossing  errors  revealed  that  on-board  driving  was  different  from  teleoperation-only  and  that 
teleoperation-only  was  different  from  teleoperation  with  cruise  control. 

The  results  for  workload  ratings  demonstrated  a  significant  finding  for  the  main  effect  of  mode, 
F(2,6)  =  5.10,  p  =  0.05,  and  rj2  =  0.630.  The  effect  size  was  large,  accounting  for  63.0%  of  the 
variance.  The  workload  rating  means  for  each  condition  with  associated  standard  error  bars  are 
presented  in  figure  8.  The  main  effect  for  speed  and  the  interaction  effect  were  both  non¬ 
significant.  The  Tukey’s  LSD  post-hoc  for  mode  test  revealed  that  the  differences  were  due  to 
cell  differences  between  the  teleoperation-only  and  teleoperation-cruise  control  modes  as  well  as 
a  nearly  significant  difference  between  manual  driving  and  teleoperation-only  mode. 

The  results  for  motion  sickness  ratings  revealed  no  significant  differences  for  either  the  main 
effects  or  the  interaction.  A  summary  of  all  ANOVA  data  is  presented  in  table  4. 
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Mean  Lane  Position  Deviation  (in.)  by 
Speed  (mph) 
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Figure  5.  Mean  lane -position  deviation  (in)  by  speed  (mph)  -  night 
operation. 
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Figure  6.  Mean  lane  road-edge  crossing  errors  by  speed  (mph)  -  night  operation. 
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Figure  7.  Mean  lane  road-edge  crossing  errors  by  control  mode  -  night  operation. 
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Figure  8.  Workload  ratings  by  control  mode  -  night  operation. 
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Table  4.  Nighttime  ANOVA  table  of  dependant  measures. 


Lateral  Drift  (in.) 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

2.94 

2 

1.47 

0.21 

0.81 

0.067 

0.07 

Speed 

91.96 

1 

91.96 

47.87 

0.006 

0.941 

0.99 

Mode  x  speed 

2.17 

2 

1.08 

1.30 

0.34 

0.302 

0.19 

Road-Edge  Error 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

6061.08 

2 

3030.54 

8.67 

0.017 

0.743 

0.82 

Speed 

10127.04 

1 

10127.04 

85.47 

0.003 

0.966 

1.00 

Mode  x  speed 

144.08 

2 

72.04 

0.34 

0.73 

0.101 

0.08 

Workload 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

1947.00 

2 

973.50 

5.10 

0.05 

0.630 

0.59 

Speed 

75.26 

1 

75.26 

1.92 

0.26 

0.391 

0.16 

Mode  x  speed 

25.13 

2 

12.56 

0.78 

0.49 

0.207 

0.13 

Motion  Sickness 

SS 

df 

MS 

F 

Sig. 

Eta2 

Power 

Mode 

21.58 

2 

10.79 

2.12 

0.20 

0.415 

0.28 

Speed 

3.37 

1 

3.37 

0.75 

0.45 

0.200 

0.10 

Mode  x  speed 

0.75 

2 

0.38 

0.57 

0.59 

0.161 

0.11 

5.3  Control  and  Display  Ratings 

Control  and  display  ratings  were  derived  from  responses  taken  using  an  MCH  scale  (Cummings 
et  al.,  2006).  The  lowest  score,  1,  represents  a  very  good  rating  requiring  no  change;  10  indicates 
an  item  that  requires  complete  redesign.  It  can  be  noted  that  the  ratings  for  all  items  usually 
differed  the  most  between  on-board  driving  and  teleoperated  conditions.  There  were  some 
differences  that  were  due  to  the  change  between  teleoperated  and  teleoperated  and  cruise  control 
conditions.  Most  of  the  increased  scores  were  in  the  range  of  “minor  but  tolerable”  to 
moderately  objectionable  deficiencies.  Anecdotally,  the  Soldiers  who  used  this  system  stated 
that  they  often  felt  fatigued  in  their  hands  and  foreanns  after  about  a  30-min  operational  period. 
This  was  due  to  the  dual  function  of  the  hand-grasp  around  the  steering  wheel:  to  steer  the 
vehicle  and  to  adjust  speed  with  brake  and  accelerator  controls  that  were  grasp-oriented  on  the 
steering  wheel.  The  brake  and  accelerator  paddles  can  be  observed  in  figure  1. 

Table  5.  Mean  (M)  and  standard  deviation  (SD)  for  control  and  display  ratings  (N  =  4)  by 
mode  and  speed  -  day  operations. 


Day 

Mode 

Speed 

Onboard 

Teleop 

Teleop-Cruise 

8  mph 

15  mph 

Controls 

Steering 

1.38,  0.5 

2.13,0.9 

2.00,  0.7 

1.83,  .08 

1.83,0.8 

Accelerator 

1.25,0.5 

3.88,  1.8 

2.63,  1.3 

2.58,  1.7 

2.58,  1.7 

Brake 

1.50,0.5 

3.38,2.1 

2.38,  1.4 

2.42,  1.7 

2.42,  1.7 

Displays 

Video 

2.00,  1.9 

2.38,0.5 

1.88,  0.6 

2.17,  1.2 

2.00,  1.1 

Speedometer 

1.50,  0.7 

3.00,  1.1 

2.25,  1.0 

2.33,  1.2 

2.17,  1.1 
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Table  6.  Mean  (M)  and  standard  deviation  (SD)  for  control  and  display  ratings  (N  =  4)  by 
mode  and  speed  -  night  operations. 


Night 

Mode 

Speed 

Onboard 

Teleop 

Teleop-Cruise 

8  mph 

15  mph 

Controls 

Steering 

1.25,  0.5 

1.75,  0.9 

1.88,  0.8 

1.58,  0.8 

1.67,  0.8 

Accelerator 

1.25,0.5 

3.38,  1.4 

3.13,  1.5 

2.67,  1.6 

2.50,  1.5 

Brake 

1.50,0.5 

3.00,  1.5 

3.13,  1.5 

2.50,  1.5 

2.58,  1.4 

Displays 

Video 

1.75,  1.5 

2.38,  1.4 

2.25,  1.2 

2.00,  1.2 

2.25,  1.3 

Speedometer 

1.50,  0.9 

2.63,  1.4 

2.25,  1.2 

2.17,  1.2 

2.08,  1.3 

6.  Discussion 


Differences  were  revealed  for  the  effect  of  vehicle  speed  only  under  day  conditions.  However, 
independent  variables  for  both  control  mode  and  vehicle  speed  yielded  significant  differences 
among  the  treatment  groups  for  performance  measures  and  workload  ratings  under  nighttime 
conditions.  The  daytime  results  highlighted  the  effects  of  vehicle  speed  differences  but  revealed 
no  significant  results  for  control  mode,  which  was  the  primary  variable  of  interest. 

The  sample  size  of  this  study  must  be  highlighted  as  a  potential  shortfall  of  this  study.  Many 
logistical  and  administrative  issues  arose  prior  to  imminent  data  collection  that  reduced  an 
already  small  sample  size  to  that  what  was  reported  in  this  study  (n  =  4).  However,  when  the 
data  were  analyzed,  moderate-to-strong  effect  sizes  in  combination  with  the  significant  or  nearly 
significant  findings  made  the  data  worth  reporting. 

The  hypothesis  regarding  performance  differences  for  the  effect  control  mode  was  not  supported 
through  the  daytime  operation  results,  however,  nighttime  results  countered  this  effect  with  a 
profoundly  stronger  treatment  effect  and  higher  resultant  statistical  power.  The  bulk  of  the 
findings  for  objective  performance  data  (lane-position  deviation  and  lane  edge  errors)  were 
significant  for  the  main  effect  of  vehicle  speed  only,  which  seems  to  be  a  natural  phenomenon 
for  almost  any  driven  system  with  a  less-than-perfect  control-display  system.  Workload, 
however,  did  show  meaningful  differences  among  control  modes  that  support  results  found  in 
previous  teleoperation  research  work  by  Scribner  and  Dahn  (2008).  These  differences  reflected  a 
relatively  low  workload  for  manned  driving  in  comparison  to  the  approximate  doubling  of 
perceived  cognitive  workload  for  the  teleoperation  mode.  Cognitive  workload  subsided 
significantly  when  the  task  of  speed  control  was  removed  from  the  teleoperation-cruise  control 
mode.  As  for  motion  sickness  scores  under  day  conditions,  no  statistical  or  practical  significance 
was  observed. 
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The  first  hypothesis  is  considered  supported  by  virtue  of  the  night  operation  data,  which  in  this 
case  provided  findings  with  high  effect  sizes,  accounting  for  a  large  portion  of  the  explained 
variance.  Night  operation  yielded  significant  findings  for  the  main  effects  of  control  mode  and 
vehicle  speed,  demonstrating  that  either  visual  sensors  and  displays  or  ambient  illumination  or  a 
combination  of  both  affected  these  treatment  conditions  significantly.  Significant  differences  for 
the  effect  of  speed  were  demonstrated  in  the  dependent  task  performance  measures  of  lane 
position  deviation  and  lane-edge  errors  as  well  as  for  cognitive  workload.  No  differences  were 
noted  for  motion  sickness.  The  lane-edge  errors  were  significant  for  both  main  effects  of  control 
mode  and  vehicle  speed.  Lane-edge  errors  paralleled  the  outcomes  for  cognitive  workload 
differences  as  there  was  an  increase  of  about  58%  error  from  manned  operation  to  teleoperated 
conditions.  Teleoperated-cruise  control  conditions  yielded  a  lower  lane-edge  error  rate  as 
compared  to  teleoperated  operation.  This  error  rate  was  less  at  35%  in  comparison  to  manned 
driving.  This  data  is  supported  by  similar  findings  by  Scribner  and  Dahn  (2008). 

Night  conditions  for  driving  using  the  night  sensor  and  display  system  in  particular  had  a 
noticeable  treatment-enhancing  effect,  thus  drastically  increasing  the  statistical  power  and 
associated  treatment  effect  sizes  that  are  reported  in  tables  3  and  4.  This  is  apparent  for  the 
number  of  significant  and  nearly  significant  findings  for  task  performance  measures  of  lane-edge 
errors  and  for  cognitive  workload  measures. 

Workload  measures  as  reported  by  the  modified  NASA-TLX  indicated  that  control  mode  was  a 
nearly  significant  and  significant  measure  for  both  day  and  night  conditions,  respectively.  Night 
conditions  again  seemed  to  have  put  more  treatment  pressure  on  the  Soldier  system,  creating 
greater  treatment  effect  sizes,  power  values,  and  significance  levels  for  this  dependent  measure. 
Under  night  conditions,  all  three  control  modes  were  significantly  different  from  each  other,  with 
associated  elevated  subjective  workload  for  teleoperated-cruise  control  over  manual  driving  and 
teleoperated  driving  over  the  other  two  operation  conditions  (figure  8). 

The  second  hypothesis  is  considered  empirically  supported  with  data  that  demonstrated  findings 
with  a  large  treatment  effect  size,  accounting  for  a  large  portion  of  the  explained  variance.  The 
second  hypothesis  of  superior  performance  for  the  manned  mode  was  also  supported  by  the  data, 
as  the  subjective  workload  for  manned  operation  mode  was  far  lower  than  the  two  teleoperated 
modes.  This  data  supports  the  previous  work  of  Dixon  et  al.  (2003),  Scribner  and  Dahn  (2008), 
and  Schipani  (2003),  who  demonstrated  that  workload  is  increased  for  higher  levels  of  operator 
involvement,  to  include  teleoperators  and  robotics  control.  The  general  notion  that  teleoperated 
and  robotic  systems  have  reduced  sensory  and  situational  information  available  and  uncoupled 
motion  effects  is  supported  in  the  general  literature  review  findings  cited  by  Chen  et  al.  (2007). 

Motion  sickness  findings  were  non-significant  and  are  considered  unsupported  by  the  evidence 
within  this  study.  This  finding  is  consistent  with  speed-related  motion  sickness  results  found  in 
an  unpublished  report  by  Scribner  (2008),  who  found  that  teleoperators  operating  from  a  moving 
HMMWV-based  platform  reported  little-to-no  motion  sickness  at  slow  speeds  of  5-15  mph  but 
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showed  marked  and  significant  increase  in  motion  sickness  in  speed  bands  from  25  to  35  mph, 
and  further  still  for  speeds  of  35  mph  and  higher.  There  were  no  night  condition  results  of  any 
value  for  motion  sickness  in  this  unpublished  study.  The  lack  of  significant  motion  sickness 
findings  is  attributed  to  the  benign  nature  of  operating  on  a  relatively  stable  test  course  at  what 
are  considered  low  speeds  and  the  data  cannot  support  nor  refute  previous  motion  sickness 
findings  (Cowings  et  al.,  1999;  Hill  and  Tauson  2005;  Kamsickas,  2003;  Schipani,  2003; 
Scribner,  2008). 

Control  and  display  ratings  were  quite  interesting  in  two  regards.  First,  these  ratings  were 
apparently  susceptible  to  change  by  both  vehicle  speed  and  control  mode  treatments.  In  all 
cases,  the  control  ratings  for  the  accelerator  and  brake  (hand  controls  used  in  this  study’s 
teleoperation  modes)  were  rated  poorer  for  both  teleoperated  conditions  than  for  standard 
controls  found  in  the  HMMWV.  This  effect  was  magnified  somewhat  under  night  conditions. 
For  example,  the  throttle  control  quality  was  rated  to  be  significantly  poorer  for  both  teleoperated 
conditions  during  the  day.  At  night,  steering,  throttle,  video  display,  and  speedometer  ratings 
were  all  rated  poorer  under  both  teleoperation  modes.  See  tables  3  and  4  for  results. 


7.  Conclusions 


The  data  seem  to  indicate  that  the  best  teleoperated  condition  for  operator  involvement  with 
simultaneous  secondary  tasks  would  be  the  teleoperated-cruise  control  condition  due  to  the 
significantly  lowered  workload  and  increased  performance  compared  to  teleoperation  without 
cruise  control.  The  data  indicate  that  if  a  Soldier  is  best  removed  from  danger  via  remote  control 
technology,  then  teleoperated  missions  should  be  performed  in  teleoperation  with  cruise  control 
mode.  The  measures  for  this  control  mode  had  less  lateral  drift  error  (vehicle  swerve),  off-road 
misses  (lane-displacement  errors),  and  lowered  subjective  cognitive  workload. 

Controls  and  displays  should  be  assessed  to  determine  what  form  and  function  changes  would  be 
beneficial  to  operator  comfort  and  endurance.  Test  participants  commented  that  the  combined 
steering-throttle -brake  system  was  good,  but  that  arm  rests  would  greatly  reduce  the  musculo¬ 
skeletal  fatigue  experienced  in  the  forearms  and  hands  while  operating  in  teleoperated  mode. 

The  dead-man  switches  (spring  return  safety  mechanisms  to  return  to  zero  input)  for  the  brake 
and  accelerator  controls  created  muscular  fatigue  as  reported  by  a  majority  of  the  test 
participants.  Force  required  to  operate  these  controls  could  be  greatly  reduced  while  still  having 
spring  return  qualities  if  released. 

Motion  sickness  data  in  this  study  were  non-significant  due  to  the  low-speed  operations  of  the 
system.  With  other  tested  teleoperated  systems,  motion  sickness  appears  to  have  significant 
effects  in  speed  bands  beyond  15  mph  (Scribner,  2008). 
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The  control  lag  within  the  system  assessed  was  not  truly  noticeable  to  the  operators  and  was  not 
considered  a  significant  factor  contributing  to  dependent  variable  differences  in  control  mode. 

The  speeds  of  operation  in  this  study  can  be  considered  “benign”  under  daytime  conditions, 
which  revealed  no  significant  differences.  However,  night  conditions,  under  which  many  Soldier 
systems  operate  in  today’s  operational  theatres,  revealed  significant  differences  in  control  mode 
for  lane-edge  contact  errors  and  cognitive  workload.  It  should  be  noted  that  night  conditions 
may  increase  treatment  effect  sizes  and  statistical  power  when  conducting  future  teleoperated 
systems  experiments.  The  differences  found  in  this  research  are  best  attributed  to  reduced 
sensory  information  availability  paired  with  slight  uncoupled  motion  effects. 

In  furthering  research  of  this  type,  it  would  be  recommended  that  the  number  of  available  test 
participants  be  increased  to  allow  sufficient  statistical  power  under  day  conditions  if  a  system  is 
to  be  operational  in  this  condition.  Additionally,  extended  and  more  realistic  operational 
scenarios  employing  up  to  two  hours  of  continuous  operations  under  day  and  night  operations 
should  be  considered.  If  safety  strictures  allow,  it  is  proposed  that  workload,  vigilance,  and 
fatigue  difference  effects  for  various  control  modes  would  be  greatly  enhanced  under  these 
conditions.  These  suggestions  would  reveal  important  human  factors  engineering  (HFE)  aspects 
that  can  be  countered  with  improved  HFE  design  suggestions  to  prolong  mission  scenario 
engagement,  reduce  workload  and  stress,  and  that  increase  Soldier  combat  performance  and 
comfort  within  such  systems. 
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Appendix  A.  Volunteer  Agreement  Affidavit 


This  appendix  appears  in  its  original  format,  without  editorial  change. 
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VOLUNTEER  AGREEMENT  AFFIDAVIT: 

ARL-HRED  Local  Adaptation  of  DA  Form  5303-R.  For  use  of  this  form,  see  AR  70-25  or  AR 

40-38 


The  proponent  for  this 

U.S.  Army  Research  Laboratory 

research  is: 

Human  Research  and  Engineering 

Directorate 

Aberdeen  Proving  Ground,  MD 

21005 

Authority: 

Privacy  Act  of  1974,  10  U.S.C.  3013,  [Subject  to  the  authority,  direction, 
and  control  of  the  Secretary  of  Defense  and  subject  to  the  provisions  of 
chapter  6  of  this  title,  the  Secretary  of  the  Army  is  responsible  for,  and  has 
the  authority  necessary  to  conduct,  all  affairs  of  the  Department  of  the 

Army,  including  the  following  functions:  (4)  Equipping  (including 
research  and  development),  44  USC  3101  [The  head  of  each  Federal 
agency  shall  make  and  preserve  records  containing  adequate  and  proper 
documentation  of  the  organization,  functions,  policies,  decisions, 
procedures,  and  essential  transactions  of  the  agency  and  designed  to 
furnish  the  information  necessary  to  protect  the  legal  and  financial  rights  of 
the  Government  and  of  persons  directly  affected  by  the  agency's  activities] 

Principal  purpose: 

To  document  voluntary  participation  in  the  Research  program. 

Routine  Uses: 

The  SSN  and  home  address  will  be  used  for  identification  and  locating 
purposes.  Information  derived  from  the  project  will  be  used  for 
documentation,  adjudication  of  claims,  and  mandatory  reporting  of 
medical  conditions  as  required  by  law.  Information  may  be  furnished  to 
Federal,  State,  and  local  agencies. 

Disclosure: 

The  furnishing  of  your  SSN  and  home  address  is  mandatory  and  necessary 
to  provide  identification  and  to  contact  you  if  future  information  indicates 
that  your  health  may  be  adversely  affected.  Failure  to  provide  the 
information  may  preclude  your  voluntary  participation  in  this  data 
collection. 

Part  A  •  Volunteer  agreement  affidavit  for  subjects  in  approved  Department  of  Army 

research  projects 

Note:  Volunteers  are  authorized  medical  care  for  any  injury  or  disease  that  is  the  direct  result  of 

participating  in  this  project  (under  the  provisions  ofAR  40-38  and  AR  70-25). 
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Part  B  •  To  be  completed  by  the  Principal  Investigator 

Note:  Instruction  for  elements  of  the  informed  consent  provided  as  detailed  explanation  in 

accordance  with 

Appendix  C,  AR  40-38  or  AR  70-25. 


Purpose  of  the  Research 

You  are  invited  to  participate  in  a  study  designed  to  evaluate  the  effects  of  different  road 


Title  of  Research  Project: 

1.  A  Comparison  of  Soldier  Performance  in  a  Moving  Command  Vehicle 
Under  Manned,  Teleoperated  and  Semi-Autonomous  Robotic  Mine 
Detector  System  Control  Modes. 

Human  Use  Protocol  Log  # 
Number: 

ARL-20098-09008 

Principal  Investigator: 

David  Scribner 

Phone:  410-278-5983 

email:  david.scribner@us.army.mil 

Associate  Investigators: 

Asi  Animashaun 

4 1 0-278-5 883,  aanimashaun@arl. army. mil 

Will  Culbertson 

254-288-9222,  will.culbertson@us.army.mil 

Location  of  Research: 

Aberdeen  Proving  Ground,  MD 

Dates  of  Participation: 

March- October  2009 

surfaces  and  speeds  on  remote  control  vehicle  performance.  More  specifically,  the  purpose  of 
this  study  is  to  compare  three  different  speeds  of  control  and  their  effects  on  route  path  quality, 
course  time,  and  ratings  of  mental  effort,  motion  sickness,  stress,  and  vehicle  controllability.  This 
study  will  be  conducted  the  Aberdeen  Test  Center  (ATC)  test  courses  at  Aberdeen  Proving 
Ground,  Maryland. 

Procedures 

Participation  in  this  study  will  require  two  weeks  of  visitation  to  the  test  course  facility  at  APG, 
MD.  On  the  first  day,  you  will  be  asked  to  (1)  provide  written  informed  consent  to  participate  in 
the  study,  (2)  choose  whether  or  not  to  provide  the  principle  investigator  your  ASVAB  scores  (3) 
be  assigned  a  confidential  participant  ID  number,  (4)  complete  a  demographics  questionnaire, 
and  (5)  be  tested  for  visual  acuity  and  color  vision.  After  this,  you  will  be  familiarized  with  the 
teleoperated  system,  the  test  courses,  and  safety  procedures  pertaining  to  the  operation  of  the 
system.  You  will  train  for  several  days  at  the  beginning  of  the  first  week  on  how  to  specifically 
operate  the  teleoperated  and  manned  systems.  During  the  second  week,  you  will  be  exposed  to 
twelve  experimental  trials,  combinations  of  manned/unmanned  operation,  maximum  speed,  and 
time  of  operation  (day  or  night  camera).  Following  this,  you  will  be  de -briefed  and  given 
information  on  how  to  contact  the  researcher  for  questions  that  you  may  have  about  your  data  or 
the  study  after  it  is  complete.  Your  time  commitment  in  this  study  will  be  approximately  from 
0800-1700  with  a  one-hour  break  for  lunch.  During  night  trials,  your  time  commitment  will  be 
approximately  from  1800-0200  with  a  one-hour  break  for  eating  or  rest  after  four  hours. 
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There  is  a  risk  of  motion  sickness  in  this  study,  as  you  will  be  moving  in  a  vehicle  that  follows 
the  teleoperated  vehicle,  at  speeds  up  to  but  not  exceeding  25  mph.  This  is  considered  a  safe 
speed,  which  may  generate  incapacitating  motion  sickness.  You  are  free  to  withdraw  from  this 
study  at  any  time  for  any  reason,  including  feeling  any  effects  of  motion  sickness.  You  will  be 
operating  inside  of  a  vehicle  that  will  be  heated  and  provide  shelter  from  weather  effects  such  as 
wind  and  precipitation.  You  will  be  asked  to  wear  a  seat  belt  at  all  times  when  operating  the 
vehicle.  You  will  also  be  asked  to  wear  a  safety  helmet  for  protection  when  inside  the  control 
vehicle. 

You  will  be  asked  to  complete  questionnaires  relating  to  workload,  stress,  and  motion  sickness  at 
mid-completion  and  at  the  end  after  each  experimental  trial.  You  will  also  be  asked  to  fill  out  a 
baseline  motion  sickness  questionnaire.  The  test  course  you  will  operate  the  vehicle  on  is 
approximately  4800  meters  in  length.  You  will  operate  one  lap  in  one  direction  and  then  another 
lap  in  the  opposite  direction.  One  run  will  be  about  a  half-hour  long.  You  will  be  permitted  to 
have  a  break  for  30  minutes  between  trials.  This  will  require  approximately  6  hours  of 
experimental  teleoperation  in  all,  not  including  training.  You  may  not  be  eligible  to  participate  in 
this  study  if:  (1)  your  visual  acuity  is  less  than  20/30  when  corrected  with  glasses  or  contact 
lenses,  or  if  2)  your  medical  profile  indicates  that  your  health  status  requires  approval  by  your 
physician. 


Benefits 

You  will  receive  no  benefits  from  participating  in  the  project,  other  than  the  personal 
satisfaction  of  supporting  research  efforts  to  better  understand  factors  that  affect  differences  in 
various  remote  control  modes  for  teleoperated  mine  detection  systems. 

Risks 

Risks  associated  with  this  evaluation  are  minimal  and  are  less  than  those  encountered  by  Soldiers 
during  their  normal  field  training  exercises  or  by  civilians  driving  on  public  roads.  There  is  a 
moderate  risk  of  motion  sickness  and  steps  will  be  taken  to  prepare  for  this  possibility.  These 
steps  include  having  a  motion  sickness  bag  and  pre-soaked  sterile  wipes  and  hand  sanitizer 
available  in  case  of  vomiting.  There  will  also  be  refreshments  offered  and  a  place  to  sit 
comfortably  or  lie  down  for  any  time  period  required.  If  you  develop  motion  sickness 
symptoms,  we  will  ask  you  to  remain  at  the  site  until  symptoms  disappear. 

Members  of  the  test  administration  staff  will  be  close  to  you  throughout  all  evaluation  trials  to 
assist  you  should  a  problem  arise.  If  you  ask  to  terminate  the  test,  care  will  be  taken  to  minimize 
risks  and  you  will  be  allowed  to  cease  participation.  You  will  have  a  break  of  at  least  10  minutes 
lasting  up  to  30  minutes  between  conditions. 

Confidentiality 

All  data  and  information  obtained  about  you  will  be  considered  privileged  and  held  in 
confidence.  We  would  like  your  permission  to  take  photos  and  audio/video  recording  of  you 
during  your  experimental  runs.  The  photos  will  be  used  for  presentation  and  a  final  report.  The 
video  may  be  used  for  a  briefing  on  the  experiment.  The  photos  will  have  your  face  and  name 
blurred  to  obscure  identifying  information.  Video  footage  will  be  taken  at  such  an  angle  that  you 
will  not  be  able  to  be  positively  identified.  Complete  confidentiality  cannot  be  promised, 
particularly  if  you  are  a  military  service  member,  because  information  bearing  on  your  health 
may  be  required  to  be  reported  to  appropriate  medical  or  command  authorities.  In  addition, 
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applicable  regulations  note  the  possibility  that  the  U.S.  Army  Human  Research  Protection  Office 
officials  may  inspect  the  records.  Please  indicate  below  if  you  will  agree  to  allow  us  to  record 
you.  You  can  still  be  in  the  study  if  you  prefer  not  to  be  recorded. 


I  give  consent  to  being  audio  taped  during  this  study  _ YES  _ NO  please  initial. _ 

I  give  consent  to  being  videotaped  during  this  study  _ YES  _ NO  please  initial. _ 

I  give  consent  to  being  photographed  during  this  study  _ YES  _ NO  please  initial _ 

Disposition  of  Volunteer  Agreement  Affidavit 

The  Principal  Investigator  will  retain  the  original  signed  Volunteer  Agreement  Affidavit  and 
forward  a  photocopy  of  it  to  the  Chair  of  the  Human  Use  Committee  after  the  data  collection. 

The  Principal  Investigator  will  provide  a  copy  of  the  signed  and  initialed  Affidavit  to  you. 

Obtaining  of  ASVAB  Scores 

IF  YOU  ARE  AN  ACTIVE  DUTY  ENLISTED  MILITARY  VOLUNTEER ,  we  would  like  to 
obtain  your  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  scores  for  potential  data 
analysis.  The  ASVAB  scores  would  be  used  strictly  for  research  purposes.  The  results  of  any 
such  analyses  would  be  presented  for  the  group  of  participants  as  a  whole;  and  no  names  will  be 
used.  With  your  permission,  we  will  obtain  these  scores  by  sending  a  copy  of  this  signed  consent 
form  along  with  your  Social  Security  Number  to  the  Defense  Manpower  Data  Center  (DMDC)  in 
Seaside,  CA  where  ASVAB  scores  may  be  obtained  from  their  databases  in  Arlington,  VA  or 
Seaside,  CA.  If  you  do  not  wish  your  ASVAB  scores  to  be  released  to  the  principal  investigator, 
you  will  still  be  allowed  to  participate  in  the  research. 

If  you  would  like  to  participate  in  this  research,  please  sign  one  of  the  following  statements,  and 
then  complete  the  information  requested  at  the  end  of  this  form: 

I  DO  AUTHORIZE  you  to  obtain  my  ASVAB  scores. _ 

(Your  Signature) 

I  DO  NOT  AUTHORIZE  you  to  obtain  my  ASVAB  scores. 

(Your  Signature) 

Contacts  for  Additional  Assistance 

If  you  have  questions  concerning  your  rights  on  research-related  injury,  or  if  you  have  any 
complaints  about  your  treatment  while  participating  in  this  research,  you  can  contact: 

Chair,  Human  Use  Committee  OR  Office  of  the  Chief  Counsel 

U.S.  Army  Research  Laboratory  U.S.  Army  Research  Laboratory 

Human  Research  and  Engineering  2800  Powder  Mill  Road 

Directorate 

Aberdeen  Proving  Ground,  MD  21005  Adelphi,  MD  20783-1197 
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(410)  278-5992  or  (DSN)  298-5992 


(301)  394-1070  or  (DSN)  290-1070 


I  do  hereby  volunteer  to  participate  in  the  research  project  described  in  this  document.  I  have  full 
capacity  to  consent  and  have  attained  my  18th  birthday.  The  implications  of  my  voluntary 
participation,  duration,  and  purpose  of  the  research  project,  the  methods  and  means  by  which  it  is 
to  be  conducted,  and  the  inconveniences  and  hazards  that  may  reasonably  be  expected  have  been 
explained  to  me.  I  have  been  given  an  opportunity  to  ask  questions  concerning  this  research 
project.  Any  such  questions  were  answered  to  my  full  and  complete  satisfaction.  Should  any 
further  questions  arise  concerning  my  rights  or  project  related  injury,  I  may  contact  the  ARL- 
HRED  Human  Use  Committee  Chairperson  at  Aberdeen  Proving  Ground,  Maryland,  USA 
by  telephone  at  410-278-5992  or  DSN  298-5992. 1  understand  that  any  published  data  will  not 
reveal  my  identity.  If  I  choose  not  to  participate,  or  later  wish  to  withdraw  from  any  portion  of  it, 
I  may  do  so  without  penalty.  I  understand  that  military  personnel  are  not  subject  to  punishment 
under  the  Uniform  Code  of  Military  Justice  for  choosing  not  to  take  part  as  human  volunteers 
and  that  no  administrative  sanctions  can  be  given  me  for  choosing  not  to  participate.  I  may  at  any 
time  during  the  course  of  the  project  revoke  my  consent  and  withdraw  without  penalty  or  loss  of 
benefits.  However,  I  may  be  required  (military  volunteer)  or  requested  (civilian  volunteer)  to 
undergo  certain  examinations  if,  in  the  opinion  of  an  attending  physician,  such  examinations  are 
necessary  for  my  health  and  well  being. 


Printed  Name  Of  Volunteer  (First,  ML,  Last) 

Social  Security  Number 
(SSN) 

Permanent  Address  Of  Volunteer 

Date  Of  Birth 
(Month,  Day,  Year) 

Today’s  Date 
(Month,  Day,  Year) 

Signature  Of  Volunteer 

Signature  Of  Administrator 
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Appendix  B.  Demographic  Data  Collection  Form 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


DEMOGRAPHICS  AND  EXPERIENCE  QUESTIONNAIRE  Soldier  ID _ 

Age _ Height _ ft _ in  Weight _ lbs 

Rank _ Date  entered  military  (month)_ _ (year)_ _ 

Date  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  taken  (month) _ 

(Year). _ N/A _ 

If  applicable,  Primary  MOS. _  Secondary  MOS_ _  OR  Job 

Title_ _ 

1.  Have  you  operated  a  remote  military  system  before?  _ .Yes  _ No 

2.  Have  you  operated  radio-controlled  hobby  systems  before?  (Car,  plane)  _ _Yes  _ No 

3.  How  well  do  you  feel  you  perform  with  remote  vehicles? 

_ .Poor  _ Below  Average  _ .Average  _ Above  Average  _ .Excellent 

4.  Does  your  Military  Occupational  Specialty  include  driving  any  vehicles?  _ .Yes  _ No 

5.  Are  you  _ .left  handed  or  _ right  handed? 

6.  Do  you  use  your _ _ left  eye  or _ _ .right  eye  to  aim  a  weapon? 

7.  Do  you  wear  glasses/contact  lenses  when  you  drive  ? _ Yes  _ No 

8.  a.  Do  you  play  video  games  or  computer  games?  _ .Yes  No 

b.  What  type  of  specific  systems  do  you  use?  _ Console  _ PC  _ Both 

9.  Do  you  ever  play  simulations  or  games  that  have  driving  involved?  . _ Yes  _ No 

10.  How  well  do  you  think  you  play  driving  video  games? 

_ .Poor  _ Below  Average  _ Average  _ Above  Average  _ Excellent 

11.  What  is  your  current  education  level? 

_ High  School  _ Junior  College  _ Bachelor’s  Degree  _ MA  _ _PhD 

12.  How  susceptible  are  you  to  motion  sickness? _ Low  _ Moderate  _ high 


Vision  Testing  Score:  (Acuity). 


(Color  Vision) 
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Appendix  C.  Modified  NASA  Task  Load  Index  (M-NTLX) 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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NASA-TLX  Workload  Questionnaire  (Weighting  Selection) 


Soldier  ID _ 

Date  /  / 


(one  time  only,  per  test  participant) 


For  each  pair,  select  the  one  element  that  is 
more  important  for  measurement  of  workload 
for  the  task. 


Mental  Demand 

/ 

Physical  Demand 

n 

Mental  Demand 

/ 

Temporal  Demand 

□ 

□ 

Physical  Demand 

/ 

Temporal  Demand 

□ 

NASA-TLX  RATING  SCALE  DEFINITIONS 

MENTAL  DEMAND  Low/High  How  much  mental  and  perceptual  activity 

was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

PHYSICAL  DEMAND  L  o  w/High  How  much  physical  activity  was  required 

(e.g..  pushing,  pulling,  turning,  controlling,  activating,,  etc.)? 
Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or 
strenuous  restful  or  laborious? 


TEMPORAL  DEMAND  L  o  w/Hig/z  How  much  time  pressure  did  you  feel  due  to 

the  rate  or  pace  at  which  the  tasks  or  task  elements  occurred? 
Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 
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NASA-TLX  Workload  Questionnaire  (Weighting  Selection) 


Soldier  ID _ 

Date. _ 

C  ondition _ 

For  each  workload  element  listed  below,  please  indicate  (with  an  exact  mark  on  the  line)  how  much 

each  element  contributed  to  your  overall  workload  experienced  in  the  task  you  just  performed.  Please  write  the 

corresponding  number  for  your  mark  in  the  space  provided  below  each  line. 


Mental  Workload 


Physical  demand 


Temporal  Demand 


0 

100 

Low 

1  1 

What  number  is  this? 

1  1  1  1  1  1  1  1  1  1  1  1 

High 

1  1  1  1  1  1  1 

0 

100 

Low  High 


What  number  is  this?. 


0 

100 

Low 

What  number  is  this? 

High 
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Intentionally  lelt  blank. 
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Appendix  D.  Motion  Sickness  Assessment  Questionnaire  (MSAQ)  Form 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Motion  Sickness  Assessment  Questionnaire 


Soldier  ID 


Date _ 

Condition: _ 

Using  the  scale  below,  please  circle  the  number  that  rates  how  accurately  the  following  statements  describe  your 
experience. 

Not  at  all  Severely 


1 . 1  felt  sick  to  my  stomach 

O  T  faint  1 1 1/-4* 

1 - 2- 

Not  at  all 

1 _ 2 _ 

-—3- 

___3 _ 

- 4— 

_ 4 _ 

-----5- 

___S— 

- 6_ 

_ fi _ 

- 7. 

_ 7 _ 

- 8- 

_ g _ 

- 9 
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